This invention relates to an automated test tool for computer systems and in particular to a test automation tool having dynamic attributes for testing a computing environment.
An important aspect of design and fabrication of any technology is the testing of different features that make up that technology. Testing is not only used for quality control purposes, to ensure the reliability of existing technology, but is also a key requirement when introducing new features or when repackaging existing features in new and improved ways. This is particularly true in the computer industry as it relates to the testing of a computing environment having both hardware and software elements.
A foremost objective in providing testing tools and methodology, especially in the computer industry, is to provide thorough testing. Complex computing environments, which vary in numbers of and types of hardware and software, as well as operating systems and middleware, require many attributes to be tested to ensure system integrity. Systems have become so complex that it is frequently impossible to test all attributes of system organization and function. A challenge of present test methodologies is to be able to identify and test attributes of the system that are most likely to fail, and those that are most critical to its operation. Since system attributes can vary over a range of values, testing sufficient combinations of attributes having different values is difficult without some form of automation. However, test automation as provided by the prior art, has its own set of challenges.
A disadvantage of prior art test automation techniques is that they only test one sub-environment at a time, where attributes and their values are well defined and known. In complex computing environments where the environment includes many applications, configurations and sub-environments, test tools adaptable across the environment are not commonly available. Testing in one sub-environment or configuration alone is undesirable because it does not validate similar testing in other sub-environments or configurations.
In simple computing environments, testing is traditionally accomplished by entering the attributes and values as parameters manually, for example, on a command line for a particular test tool or script. With automation, programs can be designed to test a more inclusive set of values than those permitted by manual testing. The values to be tested in an automation scheme are normally kept in an input file. Automated test tools, therefore, have to be developed beforehand in order to make the input file available at time of test that has the test values to be used. This makes the changing of or incorporation of new attributes into the prior art automated test scenarios very difficult.
In addition, the necessity of choosing values beforehand leads to the problem of not knowing exactly what values need to be included for automated testing. Selecting particular attributes to be varied in different test scenarios for a complex computing environment could lead to an endless number of variations. The prior art automation schemes only cover limited predefined attributes and values, making it difficult to decide upon the right amount of attributes to be tested beforehand in order to ensure accurate test results.
There is a perennial need for automation tools that allow user input to customize a test scenario to particular feature of the system under test. This is because it could be impossible to account for all testing scenarios ahead of time. Often an unanticipated condition or unaccounted for need arises because of certain conditions only occurring during the test. For example, a particular test configuration may be only available in a certain time frame for testing purposes. However, in such instances, there may not be enough time and resources available to alter the test using the changed attributes, such that testing must proceed with pre-existing scenarios that do not adequately cover the computing environment.
Another problem with the existing prior art is that automated test tools are not designed to be flexible. Traditional automation tools only provide for a fixed set of input and output values. Changing a test scenario to expect different parameters, and checking for different results or errors, requires stopping tests at run-time. New input must be manually created before the test is restarted. These limitations hinder the purpose of the automation tool. Another problem relates to requirements that are hard-coded in scripts or in binary formats. In such cases, automated test tools that generate job conditions at run time do not allow the user to change pre-fixed attributes of control commands or to add or remove control commands unless they are defined in the scripts and program source files. This prevents last minute tests from being included, even when they are deemed essential.
Consequently, the designers of test tools for computing environments face many challenges today. Manual testing is tedious and impractical due to the number of test conditions that need to be varied. On the other hand, automation is inflexible under the prior art techniques and cannot be performed adequately across multiple configurations and sub-environments in a complex computing environment. Furthermore, significant time and resource constraints prevent testing of all attribute variations in such an environment. In addition, deciding what to test beforehand is often unfeasible and impractical under the current prior art techniques. Foregoing thorough testing, however, is also not a desirable option as it may lead to unwanted problems and increased risks to product performance. A new test automation tool and methodology is needed to address the above-mentioned concerns.